Papers with audio generation
UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models (2026.acl-demo)
Copied to clipboard
Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu
| Challenge: | Existing evaluation frameworks for audio foundation models are heavily reliant on English, making it difficult to objectively assess models’ performance on Chinese. |
| Approach: | They propose a unified framework that supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards. |
| Outcome: | The proposed framework supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards. |
MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling (2026.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that MAViS achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |
| Approach: | They propose an end-to-end multi-agent collaborative framework for long-sequence video storytelling that orchestrates specialized agents across multiple stages. |
| Outcome: | The proposed framework achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |
XMAD-Bench: Cross-Domain Multilingual Audio Deepfake Benchmark (2026.findings-eacl)
Copied to clipboard
Ioan-Paul Ciobanu, Andrei-Iulian Hîji, Nicolae Catalin Ristea, Paul Irofti, Cristian Rusu, Radu Tudor Ionescu
| Challenge: | Recent advances in audio generation led to an increasing number of deepfakes . however, these methods are typically tested in an in-domain setup . |
| Approach: | They propose a large-scale cross-domain audio deepfake benchmark comprising 668.8 hours of real and deepfak speech. |
| Outcome: | The proposed benchmark compares audio deepfake detectors with existing methods in the wild . the results show that the proposed methods perform better in different languages than existing methods . |
Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)
Copied to clipboard
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao
| Challenge: | Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video. |
| Approach: | They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights. |
| Outcome: | The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in latent diffusion models (LDMs) have markedly enhanced text-to-audio generation, yet their iterative sampling processes impose substantial computational demands, limiting practical deployment. |
| Approach: | They propose to learn straight flow for fast simulation by using flashAudio with rectified flows and immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment. |
| Outcome: | The proposed method can learn straight flow for fast simulations and reduce noise distribution. |